Skip to content

chore: add flakinmess#356

Merged
vinitkumar merged 2 commits into
masterfrom
feat/add-flakiness
Jul 22, 2026
Merged

chore: add flakinmess#356
vinitkumar merged 2 commits into
masterfrom
feat/add-flakiness

Conversation

@vinitkumar

@vinitkumar vinitkumar commented Jul 22, 2026

Copy link
Copy Markdown
Owner

Summary by Sourcery

Integrate pytest-flakiness reporting into the Python test workflow while tightening GitHub Actions permissions and documenting the new CI scope.

Enhancements:

  • Configure the Python test job with scoped OIDC permissions and FLAKINESS_PROJECT environment for flakiness reporting.
  • Add pytest-flakiness to the CI test dependencies to enable automated flakiness detection in the pytest matrix.

Documentation:

  • Update architecture documentation to describe the Python test job's id-token usage and FLAKINESS_PROJECT scoping for the flakiness reporter.

@sourcery-ai

sourcery-ai Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor
Reviewer's guide (collapsed on small PRs)

Reviewer's Guide

Adds pytest-flakiness support to the Python CI workflow by scoping OIDC permissions and flakiness-related environment to the test job only, and updates the architecture docs to describe the new permissions model for the test matrix.

Sequence diagram for test job running pytest with pytest-flakiness via GitHub OIDC

sequenceDiagram
  actor Developer
  participant GitHubActions as GitHubActionsWorkflow
  participant TestJob as PythonTestJob
  participant Pytest as pytest
  participant Flakiness as pytest_flakiness
  participant OIDC as GitHubOIDC
  participant Service as FlakinessService

  Developer->>GitHubActions: push to repository
  GitHubActions->>TestJob: start test job (id-token: write, FLAKINESS_PROJECT)
  TestJob->>Pytest: run pytest
  Pytest->>Flakiness: load pytest-flakiness plugin
  Flakiness->>OIDC: request id-token for FLAKINESS_PROJECT
  OIDC-->>Flakiness: return OIDC id-token
  Flakiness->>Service: submit flakiness report with id-token
  Service-->>Flakiness: acknowledge report
  Flakiness-->>Pytest: mark test results with flakiness data
  Pytest-->>TestJob: exit with test and flakiness results
Loading

File-Level Changes

Change Details Files
Scope GitHub OIDC permissions and flakiness-related environment variables to the Python test job and install pytest-flakiness in CI.
  • Add contents: read and id-token: write permissions to the test job so flakiness reporting can use GitHub OIDC without broadening other jobs’ scopes.
  • Define FLAKINESS_PROJECT environment variable on the test job to identify the project for flakiness reporting.
  • Extend the dependency installation step to include pytest-flakiness alongside existing pytest tooling.
.github/workflows/pythonpackage.yml
Update architecture documentation to explain the test job’s id-token permission and flakiness configuration.
  • Describe that the Python test job now scopes id-token: write and FLAKINESS_PROJECT specifically to the pytest matrix for the flakiness reporter.
  • Clarify that these permissions are not exposed to the lint job while maintaining the existing description of workflow permissions and CI behavior.
lat.md/architecture.md

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 1 issue, and left some high level feedback:

  • Consider deriving FLAKINESS_PROJECT from the repository context (e.g., github.repository) rather than hardcoding vinitkumar/json2xml so the workflow stays accurate if the repo is renamed or forked.
  • The architecture note says the test job "scopes id-token: write and FLAKINESS_PROJECT" to the pytest matrix, but FLAKINESS_PROJECT is an environment variable rather than a permission; you might reword that sentence to avoid conflating env vars with permissions.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- Consider deriving `FLAKINESS_PROJECT` from the repository context (e.g., `github.repository`) rather than hardcoding `vinitkumar/json2xml` so the workflow stays accurate if the repo is renamed or forked.
- The architecture note says the test job "scopes `id-token: write` and `FLAKINESS_PROJECT`" to the pytest matrix, but `FLAKINESS_PROJECT` is an environment variable rather than a permission; you might reword that sentence to avoid conflating env vars with permissions.

## Individual Comments

### Comment 1
<location path="lat.md/architecture.md" line_range="79" />
<code_context>
+The `.github/workflows/` files declare the minimum `permissions:` scopes needed by each workflow, with CodeQL retaining `security-events: write` for result upload and TestPyPI retaining `id-token: write` for explicit trusted-publishing runs. The Python test job also scopes `id-token: write` and `FLAKINESS_PROJECT` to its pytest matrix so the flakiness reporter can authenticate through GitHub OIDC without exposing that permission to lint. Release-branch pushes build distributions and run Twine checks; TestPyPI upload is a manual opt-in because that external registry requires separate publisher configuration. Action references are pinned to immutable commits with the upstream tag retained in comments for reviewability, and `.github/dependabot.yml` checks the `github-actions` ecosystem weekly so those pins do not silently age. The Python test matrix pins its PyPy 3.11 job to an explicit PyPy release so CI keeps exercising the intended CPython 3.11.15-compatible runtime instead of silently drifting with runner cache updates. It also exercises regular CPython 3.15.0b3 while leaving that beta's free-threaded builds out of CI until the runner support is less brittle.
</code_context>
<issue_to_address>
**issue (typo):** Fix the subject–verb agreement in the phrase about release-branch pushes and Twine checks.

In the sentence beginning "Release-branch pushes build distributions and run Twine checks", "pushes" is singular while "run" is plural. Please align the verbs, e.g. "Release-branch pushes build distributions and runs Twine checks."

```suggestion
The `.github/workflows/` files declare the minimum `permissions:` scopes needed by each workflow, with CodeQL retaining `security-events: write` for result upload and TestPyPI retaining `id-token: write` for explicit trusted-publishing runs. The Python test job also scopes `id-token: write` and `FLAKINESS_PROJECT` to its pytest matrix so the flakiness reporter can authenticate through GitHub OIDC without exposing that permission to lint. Release-branch pushes build distributions and runs Twine checks; TestPyPI upload is a manual opt-in because that external registry requires separate publisher configuration. Action references are pinned to immutable commits with the upstream tag retained in comments for reviewability, and `.github/dependabot.yml` checks the `github-actions` ecosystem weekly so those pins do not silently age. The Python test matrix pins its PyPy 3.11 job to an explicit PyPy release so CI keeps exercising the intended CPython 3.11.15-compatible runtime instead of silently drifting with runner cache updates. It also exercises regular CPython 3.15.0b3 while leaving that beta's free-threaded builds out of CI until the runner support is less brittle.
```
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

Comment thread lat.md/architecture.md Outdated
Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com>
@vinitkumar
vinitkumar merged commit c4a2ad7 into master Jul 22, 2026
42 of 44 checks passed
@codecov

codecov Bot commented Jul 22, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (77ff48d) to head (a0554e9).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff            @@
##            master      #356   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files            7         7           
  Lines          833       833           
=========================================
  Hits           833       833           
Flag Coverage Δ
unittests 100.00% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant